In recent years, an apparatus having a camera is used for detecting a movement of the apparatus itself (i.e., an apparatus movement) or a movement of a moving body that carries the apparatus based on an image captured by the camera. The image captured by the camera is further analyzed for detecting three-dimensional information of an object in the image such as a position and a shape based on the apparatus movement and a movement of the object captured in the image (Refer to, for example, U.S. Pat. No. 6,535,114).
The apparatus movement and/or the movement of the object are, in general, detected by calculating an optical flow that vectorially represents a movement of an identical characteristic point in a sequence of successive images.
The optical flow reflects, by definition, the apparatus movement when the characteristic point is standing still, and reflects a combination of the apparatus movement and the movement of the characteristic point when the characteristic point is moving.
The camera on the moving body for detecting an obstacle is positioned to have a light axis aligned substantially in a horizontal direction for capturing a wide range of a field image in front of the moving body. The moving body may practically be an automotive vehicle or the like.
The apparatus movement, or the movement of the camera in reality, is typically detected by using the characteristic point on a surface of a road, because the characteristic point on the road is generally large in the captured image and is standing still when the characteristic point is carefully chosen. That is, the apparatus movement is detected based on the optical flow derived by tracing the characteristic point in the sequence of the successive images.
However, the camera positioned in the above-described manner with its light axis substantially aligned in a horizontal direction has a large pitch angle α about a value of 90 degrees to a vertically downward direction. Therefore, the optical flow derived by tracing the characteristic point of the road has a large detection error, thereby deteriorating the accuracy of the apparatus movement.
When the light axis of the camera has the pitch angle α as shown in FIG. 12, an imaging surface G that captures a far front field in an upper half and a near front field in a lower half generates an image that includes a movement of the characteristic point on the road projected thereon in a proportionally reduced manner to a distance from the camera. That is, the same amount of the movement of the characteristic point on the road is captured as a larger movement image in the lower half of the imaging surface G when the characteristic point is positioned closer to the camera on the moving body, or is captured as a smaller movement image in the upper half of the imaging surface G when the characteristic point is positioned far off from the camera.
Therefore, the optical flow based on the movement of the characteristic point in the captured image suffers from an error that is caused by a different behavior of the characteristic point depending on a position of the characteristic point in the imaging surface G. In general, the larger the pitch angle α is, or the farther the characteristic points is from a center of the imaging surface G, the greater the error is.
The following table shows a result of a simulation that simulates a movement of a vehicle in a lateral and front-rear directions (translational movements), and in a yaw direction (a rotational movement) based on the images of the road surface captured by the camera that is positioned in the specified pitch angle α. The table compares a true value of these movements with an average of the errors in the absolute value.
LateralFront-rearYawmovementmovementmovementTrueTrueTruePitch anglevalueErrorvalueErrorvalueErrorα(m/(m/(m/(m/(deg/(deg/[deg (rad)]frame)frame)frame)frame)frame)frame)5.7 (0.1) 0.20.00.20.10.31.211 (0.2)0.20.00.20.20.31.317 (0.3)0.20.00.20.20.31.723 (0.4)0.20.00.20.20.32.629 (0.5)0.20.00.20.20.34.3
The height of the camera from the road surface is 1.5 meter (the height of the camera corresponds to a map lamp side position of a one box type vehicle where the camera is installed), and the pitch angle α of the camera to the vertically downward direction is defined in five levels of 0.1, 0.2, 0.3, 0.4, 0.5 radians with no roll angle and no yaw angle.
Resolution of the camera is 320 pixels in width by 240 pixels in height, and a frame rate is 0.2 ms/frame.
The road surface is evenly flat with a rectangular paint mark of 2 meters in width by 5 meters in length. The vehicle is assumed to be turning at a corner of the right angle toward left at a speed of 22 km in ten seconds, which simulates a slow left turn at an intersection. The image is processed by a movement detector 11a that is described later in the embodiment section of the disclosure.
In the processing of the image, the four corners of the rectangular paint mark are specified as the characteristic points, and the translational movement of the vehicle has components only in the lateral directions (i.e., x axis in FIG. 3B), the front-rear directions (i.e., y axis in FIG. 3B). Further, the rotational movement of the vehicle has components only in the yaw direction (i.e., the rotation around z axis in FIG. 3B) with no change in the components in the pitch direction and the roll direction (i.e., the rotation around x and y axes in FIG. 3B).
As shown in the above table, the error of the rotational movement in the yaw direction increases when the pitch angle α of the camera is increased. Therefore, the camera having a greater pitch angle cannot accurately detect the movement of the apparatus and the vehicle.
On the other hand, when the camera has a smaller pitch angle α (i.e., the light axis of the camera is aligned closer to the vertically downward direction) for increasing detection accuracy of the apparatus movement, the camera captures a smaller range of the front field, thereby decreasing an obstacle detection capability. That is, the obstacle detection accuracy by the camera and the apparatus movement detection accuracy by the camera are in a trade-off relationship with each other. Therefore, the apparatus described above cannot improve the apparatus movement detection accuracy and the obstacle detection accuracy at the same time in a detection operation of the three-dimensional information of an object in the field.
Further, when the vehicle is moving forward, the characteristic point on a still object in the three-dimensional field comes closer toward the vehicle from a far side, and the optical flow of the characteristic point in the camera captured image runs downward in the image from an upper side of the image toward a lower side. Therefore, when the optical flow of the characteristic point on the road is calculated by tracing the characteristic point in the image, the characteristic point is traced in a direction that flows from an image data scarce side to an image data abundant side as shown in FIG. 13.
Therefore, in this case, an increased number of the characteristic points may possibly be detected on the lower side of the image because the image data are abundant on the lower side in comparison to the upper side. However, the characteristic points on the lower side immediately disappears from the image, thereby making it impossible to trace them for detecting the optical flow.